The present invention relates to a method for generating video holograms in real time from three-dimensional image data with depth information for extending a 3D rendering graphics pipeline.
As regards the rendering process, the invention relates to the 3D rendering pipeline or graphics pipeline, which comprises the algorithms from the vectorial, mathematical description of a scene to the pixelated image on the monitor screen. The three-dimensional image data comprise depth information and usually also additional information about material and surface properties. For example, the conversion of screen coordinates into device coordinates, texturing, clipping and anti-aliasing are performed in the 3D rendering graphics pipeline. The pixelated image, which represents a two-dimensional projection of the three-dimensional scene, and which is stored in the frame buffer of a graphics adapter, comprises the pixel values for the controllable pixels of a monitor screen, for example an LC display.
As far as the generation of holographic data is concerned, the invention relates to methods and devices for the generation of hologram values for the representation of the three-dimensional scene, which is composed of object points, on light modulator means of a holographic display device.
As regards the video holograms, this invention also relates to a holographic pipeline. Complex hologram values are generated in this pipeline in the form of pixel values for a light modulator of a holographic display device.
Such a holographic display device is substantially based on the principle that a sub-hologram is defined together with each object point of the scene to be reconstructed and that the entire hologram is formed by superposition of sub-holograms, with the help of at least one light modulator means on which a scene which is divided into object points is encoded as a entire hologram and where the scene can be seen as a reconstruction from a visibility region which lies within one periodicity interval of the reconstruction of the video hologram. In general, the principle is to reconstruct mainly that wave front that would be emitted by an object into one or multiple visibility regions.
In detail, such a device is based on the principle that the reconstruction of an individual object point only requires a sub-hologram as a subset of the entire hologram encoded on the light modulator means. The holographic display device comprises at least one screen means. The screen means is either the light modulator itself where the hologram of a scene is encoded or an optical element—such as a lens or a mirror—onto which a hologram or wave front of a scene encoded on the light modulator is imaged.
The definition of the screen means and the corresponding principles for the reconstruction of the scene in the visibility region are described in other documents filed by the applicant. In documents WO 2004/044659 and WO 2006/027228, the light modulator itself forms the screen means. In document WO 2006/119760, titled “Projection device and method for holographic reconstruction of scenes”, the screen means is an optical element onto which a hologram which is encoded on the light modulator is imaged.
In document DE 10 2006 004 300.6, “Projection device for the holographic reconstruction of scenes”, the screen means is an optical element onto which a wave front of the scene encoded on the light modulator is imaged.
Document WO 2006/066919 filed by the applicant describes a method for computing video holograms.
A ‘visibility region’ is a limited region through which the observer can watch the entire reconstruction of the scene at sufficient visibility. Within the visibility region, the wave fields interfere such that the reconstructed scene becomes visible for the observer. The visibility region is located on or near the eyes of the observer. The visibility region can be moved in the x, y and z directions and is tracked to the actual observer position with the help of known position detection and tracking systems. It is possible to use two visibility regions, one for each eye. Generally, more complex arrangements of visibility regions are also possible. It is further possible to encode video holograms such that individual objects or the entire scene seemingly lie behind the light modulator for the observer.
In this document, the term ‘light modulator means’ or ‘SLM’ denotes a device for controlling intensity, colour and/or phase of light by way of switching, gating or modulating light beams emitted by one or multiple independent light sources. A holographic display device typically comprises a matrix of controllable pixels, which reconstruct object points by modifying the amplitude and/or phase of light which passes through the display panel. A light modulator means comprises such a matrix. The light modulator means may for example be an acousto-optic modulator AOM or a continuous-type modulator. One embodiment for the reconstruction of the holograms by way of amplitude modulation can take advantage of a liquid crystal display (LCD). The present invention also relates to further controllable devices which are used to modulate sufficiently coherent light into a light wave front or into a light wave contour.
The term ‘pixel’ denotes a controllable hologram pixel of the light modulator, it represents a discrete value of the hologram point and is addressed and controlled discretely. Each pixel represents a hologram point of the hologram. In the case of an LC display, a pixel is a discretely controllable display pixel. In the case of a DMD (Digital Micro-mirror Device), such as a DLP (Digital Light Processor), a pixel is a discretely controllable micro-mirror or small group of such mirrors. In the case of a continuous light modulator, a pixel is an imaginary region which represents the hologram point. In the case of a colour representation, a pixel is typically sub-divided into multiple sub-pixels, which represent the primary colours.
The term ‘transformation’ shall be construed such to include any mathematical or computational technique which is identical to or which approximates a transformation. Transformations in a mathematical sense are merely approximations of physical processes, which are described more precisely by the Maxwellian wave equations. Transformations such as Fresnel transformations or the special group of transformations which are known as Fourier transformations, describe second-order approximations. Transformations are usually represented by algebraic and non-differential equations and can therefore be handled efficiently and at high performance using known computing means. Moreover, they can be modelled precisely using optical systems.
Document WO 2006/066919 filed by the applicant describes a method for computing video holograms. It generally includes the steps of slicing the scene into section planes which are parallel to the plane of a light modulator, of transforming all section planes in the observer plane into a visibility region, and of adding them up there. Then, the added results are back-transformed into the hologram plane, where also the light modulator is disposed, thus determining the complex hologram values of the video hologram.
Document DE 10 2006 025 096 describes a method for rendering and generating video holograms in real time from image data with depth information, where a 3D rendering graphics pipeline generates pixel values for the controllable pixels of a monitor in a first mode. It is characterised in that the pipeline is extended in a switchable manner such that in a second mode complex hologram values are generated as pixel values for a spatial light modulator SLM in at least one holographic pipeline, so that simultaneously or alternatively to the usual graphic representation the spatial light modulator is controlled with the hologram values in order to modulate an incident wave field such that the three-dimensional scene is reconstructed through interference in space.
Document DE 10 2006 042 324 describes a method for the real-time generation of video holograms. That method uses the principle that the reconstruction of a single object point only requires a sub-hologram as a subset of the entire hologram which is encoded on the SLM. It is characterised in that for each object point the contributions of the sub-holograms can be retrieved from look-up tables, and that said sub-holograms are accumulated so to form an entire hologram in order to reconstruct the entire scene.
The described methods allow the hologram values to be generated at a fast pace. However, it is necessary to include the 3D rendering graphics pipeline into further considerations. The results of a 3D rendering graphics pipeline, which describes the conversion of a three-dimensional scene into pixelated image data in the form of a two-dimensional projection of the three-dimensional scene, are provided in two memory sections, namely the frame buffer and Z buffer: The frame buffer comprises the colour values or colour information, i.e. the colour map of the scene as seen by the observer. The Z buffer comprises the depth map or depth information of the scene in a normalised representation, as seen from the observer position. Those data serve as input data for the holographic pipeline, which follows in line, and which generates complex hologram values in the form of pixel values for the light modulator. A holographic transformation can only be performed when the result of the 3D rendering graphics pipeline is available completely, i.e. when all pixelated image data are written to the frame buffer and Z buffer. The entire three-dimensional scene must be rendered by the 3D rendering graphics pipeline and the pixelated image data must be generated completely before a holographic transformation becomes possible. The mentioned pipelines are performed sequentially, i.e. one after another, which results in a disadvantageously long computing time. As a result of the long computing time, video sequences can for example not be provided with the desired refresh frequency.
However, as in conventional video technologies, a high refresh rate is desired and indispensable when displaying video holograms. Moreover, because of the great number of required computing steps, the generation of the video holograms causes a great computational load and requires high-performance and costly computing units.